Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP Adding sevriceTemplates to ManagedCluster #11

Merged
merged 25 commits into from
Oct 2, 2024

Conversation

wahabmk
Copy link
Owner

@wahabmk wahabmk commented Sep 20, 2024

Description

This PR:

  • Implements https://github.com/Mirantis/hmc/issues/ 270.
  • Adds a list of services to the ManagedCluster object, where each service corresponds to a ServiceTemplate.
  • Reconciles the services to be deployed on the target cluster via Sveltos ClusterProfile object.
  • Does not update the status of the ManagedCluster object. That will be done in a follow-up PR while working on https://github.com/Mirantis/hmc/issues/ 361.

Testing

  • Ran make dev-apply && make dev-creds-apply and waited for everything to be running.
  • Ran make dev-mcluster-apply and waited for everything to be running.

Provisioning

On Management Cluster

We can see the ClusterProfile object was created with kyverno and ingress-nginx services:

~ kubectl get clusterprofiles                          
NAME                      AGE
hmc-system-wali-aws-dev   6m46s
➜  ~~~ kubectl get clusterprofile hmc-system-wali-aws-dev -o yaml
➜  ~ kubectl get clusterprofile hmc-system-wali-aws-dev -o yaml
apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  creationTimestamp: "2024-09-20T18:48:20Z"
  finalizers:
  - clusterprofilefinalizer.projectsveltos.io
  generation: 1
  labels:
    hmc.mirantis.com/managed: "true"
    projectsveltos.io/cluster-name: wali-aws-dev
    projectsveltos.io/cluster-profile-name: hmc-system-wali-aws-dev
    projectsveltos.io/cluster-type: Capi
  name: hmc-system-wali-aws-dev
  ownerReferences:
  - apiVersion: hmc.mirantis.com/v1alpha1
    kind: ManagedCluster
    name: wali-aws-dev
    uid: f5bc78c8-46ea-4efd-bf1c-e2c3244312ff
  resourceVersion: "3632"
  uid: ccfb11a4-3f3e-4863-90a1-202e5deb8788
spec:
  clusterSelector:
    matchLabels:
      helm.toolkit.fluxcd.io/name: wali-aws-dev
      helm.toolkit.fluxcd.io/namespace: hmc-system
  continueOnConflict: false
  helmCharts:
  - chartName: kyverno
    chartVersion: 3.2.6
    helmChartAction: Install
    . . .
    registryCredentialsConfig:
      plainHTTP: true
    releaseName: kyverno
    releaseNamespace: kyverno
    repositoryName: kyverno
    repositoryURL: oci://hmc-local-registry:5000/charts
  - chartName: ingress-nginx
    chartVersion: 4.11.0
    helmChartAction: Install
    . . .
    registryCredentialsConfig:
      plainHTTP: true
    releaseName: ingress-nginx
    releaseNamespace: ingress-nginx
    repositoryName: ingress-nginx
    repositoryURL: oci://hmc-local-registry:5000/charts
    values: |
      fullnameOverride: ingress-nginx
  reloader: false
  stopMatchingBehavior: WithdrawPolicies
  syncMode: Continuous
  tier: 100
status:
  matchingClusters:
  - apiVersion: cluster.x-k8s.io/v1beta1
    kind: Cluster
    name: wali-aws-dev
    namespace: hmc-system

We can see the associated ClusterSummary object was also created and reports that the services have been "Provisioned" onto the target cluster:

~ kubectl -n hmc-system get clustersummary hmc-system-wali-aws-dev-capi-wali-aws-dev -o yaml
apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterSummary
metadata:
  . . .
  generation: 1
  . . .
  resourceVersion: "6030"
  uid: 500b3a9b-8bf7-479a-a1e9-7edaf314e9dc
spec:
  clusterName: wali-aws-dev
  clusterNamespace: hmc-system
  clusterProfileSpec:
    clusterSelector:
      matchLabels:
        helm.toolkit.fluxcd.io/name: wali-aws-dev
        helm.toolkit.fluxcd.io/namespace: hmc-system
    continueOnConflict: false
    helmCharts:
    - chartName: kyverno
      chartVersion: 3.2.6
      helmChartAction: Install
      . . .
      registryCredentialsConfig:
        plainHTTP: true
      releaseName: kyverno
      releaseNamespace: kyverno
      repositoryName: kyverno
      repositoryURL: oci://hmc-local-registry:5000/charts
    - chartName: ingress-nginx
      chartVersion: 4.11.0
      helmChartAction: Install
      . . .
      registryCredentialsConfig:
        plainHTTP: true
      releaseName: ingress-nginx
      releaseNamespace: ingress-nginx
      repositoryName: ingress-nginx
      repositoryURL: oci://hmc-local-registry:5000/charts
      values: |
        fullnameOverride: ingress-nginx
    reloader: false
    stopMatchingBehavior: WithdrawPolicies
    syncMode: Continuous
    tier: 100
  clusterType: Capi
status:
  dependencies: no dependencies
  featureSummaries:
  - featureID: Helm
    hash: 8ZDFC0FQZ2j1VHZPAeecZpAtyOUBotyUGBJGosO4tYA=
    lastAppliedTime: "2024-09-20T18:53:51Z"
    status: Provisioned
  helmReleaseSummaries:
  - releaseName: kyverno
    releaseNamespace: kyverno
    status: Managing
    valuesHash: Eq4yyx7ALQHto1gbEnwf7jsNxTVy7WuvI5choD2C4SY=
  - releaseName: ingress-nginx
    releaseNamespace: ingress-nginx
    status: Managing
    valuesHash: qYgUi/xTJIMlaXCLxb/XjCBv5xso8nVHHQ0copZdxl4=

On Target Cluster

We can see both kyverno and ingress-nginx running on the target cluster:

~ kubectl get pod -A | grep Running
ingress-nginx    ingress-nginx-controller-5bfc858768-m5xd4        1/1     Running   0          3m53s
kube-system      aws-cloud-controller-manager-fjfg2               1/1     Running   0          6m10s
kube-system      calico-kube-controllers-695f6448bd-fckbc         1/1     Running   0          7m7s
kube-system      calico-node-7tv5t                                1/1     Running   0          6m49s
kube-system      calico-node-wkxvg                                1/1     Running   0          4m53s
kube-system      coredns-6997b8f8bd-f966x                         1/1     Running   0          4m43s
kube-system      coredns-6997b8f8bd-ht4qs                         1/1     Running   0          4m43s
kube-system      ebs-csi-controller-5c9db44f4f-5cs6w              5/5     Running   0          7m4s
kube-system      ebs-csi-controller-5c9db44f4f-6twcq              5/5     Running   0          7m4s
kube-system      ebs-csi-node-ctcfp                               3/3     Running   0          6m49s
kube-system      ebs-csi-node-mh8w2                               3/3     Running   0          4m53s
kube-system      kube-proxy-gsw28                                 1/1     Running   0          6m49s
kube-system      kube-proxy-wkz7d                                 1/1     Running   0          4m53s
kube-system      metrics-server-7cc78958fc-n6jrp                  1/1     Running   0          7m7s
kyverno          kyverno-admission-controller-776987899-n9mt9     1/1     Running   0          6m50s
kyverno          kyverno-background-controller-86b9f95c96-bbnmk   1/1     Running   0          6m50s
kyverno          kyverno-cleanup-controller-7bbfc97569-5hjtn      1/1     Running   0          6m50s
kyverno          kyverno-reports-controller-665ccb5b65-cvb6d      1/1     Running   0          6m50s
projectsveltos   sveltos-agent-manager-67d6ffbd86-5vx9z           1/1     Running   0          6m57s

Setting install=false for ingress-nginx (scroll horizontally)

MANAGEMENT CLUSTER TARGET CLUSTER

By setting install=false on the ManagedCluster object, the ingress-nginx service was removed from ClusterProfile -> ClusterSummary objects:

~ kubectl -n hmc-system get clustersummary hmc-system-wali-aws-dev-capi-wali-aws-dev -o yaml
apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterSummary
metadata:
  . . .
  generation: 2
  . . .
  resourceVersion: "8661"
  uid: 500b3a9b-8bf7-479a-a1e9-7edaf314e9dc
spec:
  clusterName: wali-aws-dev
  clusterNamespace: hmc-system
  clusterProfileSpec:
    clusterSelector:
      matchLabels:
        helm.toolkit.fluxcd.io/name: wali-aws-dev
        helm.toolkit.fluxcd.io/namespace: hmc-system
    continueOnConflict: false
    helmCharts:
    - chartName: kyverno
      chartVersion: 3.2.6
      helmChartAction: Install
      . . .
      registryCredentialsConfig:
        plainHTTP: true
      releaseName: kyverno
      releaseNamespace: kyverno
      repositoryName: kyverno
      repositoryURL: oci://hmc-local-registry:5000/charts
    reloader: false
    stopMatchingBehavior: WithdrawPolicies
    syncMode: Continuous
    tier: 100
  clusterType: Capi
status:
  dependencies: no dependencies
  featureSummaries:
  - featureID: Helm
    hash: 2BR25VJae9DRUoGqxh8+6vQ+pKRWoogtkqCCJajwtek=
    lastAppliedTime: "2024-09-20T19:00:12Z"
    status: Provisioned
  helmReleaseSummaries:
  - releaseName: kyverno
    releaseNamespace: kyverno
    status: Managing
    valuesHash: Eq4yyx7ALQHto1gbEnwf7jsNxTVy7WuvI5choD2C4SY=

We can see ingress-nginx was installed from target cluster.

~ kubectl get pod -A | grep Running
kube-system      aws-cloud-controller-manager-fjfg2                         1/1     Running     0          9m45s
kube-system      calico-kube-controllers-695f6448bd-fckbc                   1/1     Running     0          10m
kube-system      calico-node-7tv5t                                          1/1     Running     0          10m
kube-system      calico-node-wkxvg                                          1/1     Running     0          8m28s
kube-system      coredns-6997b8f8bd-f966x                                   1/1     Running     0          8m18s
kube-system      coredns-6997b8f8bd-ht4qs                                   1/1     Running     0          8m18s
kube-system      ebs-csi-controller-5c9db44f4f-5cs6w                        5/5     Running     0          10m
kube-system      ebs-csi-controller-5c9db44f4f-6twcq                        5/5     Running     0          10m
kube-system      ebs-csi-node-ctcfp                                         3/3     Running     0          10m
kube-system      ebs-csi-node-mh8w2                                         3/3     Running     0          8m28s
kube-system      kube-proxy-gsw28                                           1/1     Running     0          10m
kube-system      kube-proxy-wkz7d                                           1/1     Running     0          8m28s
kube-system      metrics-server-7cc78958fc-n6jrp                            1/1     Running     0          10m
kyverno          kyverno-admission-controller-776987899-n9mt9               1/1     Running     0          10m
kyverno          kyverno-background-controller-86b9f95c96-bbnmk             1/1     Running     0          10m
kyverno          kyverno-cleanup-controller-7bbfc97569-5hjtn                1/1     Running     0          10m
kyverno          kyverno-reports-controller-665ccb5b65-cvb6d                1/1     Running     0          10m
projectsveltos   sveltos-agent-manager-67d6ffbd86-5vx9z                     1/1     Running     0          10m

Making services list empty

MANAGEMENT CLUSTER TARGET CLUSTER

We see that the ClusterSummary object does not show any helmCharts list:

~ kubectl -n hmc-system get clustersummary hmc-system-wali-aws-dev-capi-wali-aws-dev -o yaml
apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterSummary
metadata:
  creationTimestamp: "2024-09-20T18:48:20Z"
  finalizers:
  - clustersummaryfinalizer.projectsveltos.io
  generation: 3
  labels:
    hmc.mirantis.com/managed: "true"
    projectsveltos.io/cluster-name: wali-aws-dev
    projectsveltos.io/cluster-profile-name: hmc-system-wali-aws-dev
    projectsveltos.io/cluster-type: Capi
  name: hmc-system-wali-aws-dev-capi-wali-aws-dev
  namespace: hmc-system
  ownerReferences:
  - apiVersion: config.projectsveltos.io/v1beta1
    kind: ClusterProfile
    name: hmc-system-wali-aws-dev
    uid: ccfb11a4-3f3e-4863-90a1-202e5deb8788
  resourceVersion: "9610"
  uid: 500b3a9b-8bf7-479a-a1e9-7edaf314e9dc
spec:
  clusterName: wali-aws-dev
  clusterNamespace: hmc-system
  clusterProfileSpec:
    clusterSelector:
      matchLabels:
        helm.toolkit.fluxcd.io/name: wali-aws-dev
        helm.toolkit.fluxcd.io/namespace: hmc-system
    continueOnConflict: false
    reloader: false
    stopMatchingBehavior: WithdrawPolicies
    syncMode: Continuous
    tier: 100
  clusterType: Capi
status:
  dependencies: no dependencies
  featureSummaries:
  - featureID: Helm
    hash: 47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=
    lastAppliedTime: "2024-09-20T19:02:31Z"
    status: Provisioning

As expected, we can see that both ingress-nginx and kyverno have been uninstalled from the target cluster:

~ kubectl get pod -A  
NAMESPACE        NAME                                       READY   STATUS    RESTARTS   AGE
kube-system      aws-cloud-controller-manager-fjfg2         1/1     Running   0          15m
kube-system      calico-kube-controllers-695f6448bd-fckbc   1/1     Running   0          16m
kube-system      calico-node-7tv5t                          1/1     Running   0          15m
kube-system      calico-node-wkxvg                          1/1     Running   0          13m
kube-system      coredns-6997b8f8bd-f966x                   1/1     Running   0          13m
kube-system      coredns-6997b8f8bd-ht4qs                   1/1     Running   0          13m
kube-system      ebs-csi-controller-5c9db44f4f-5cs6w        5/5     Running   0          15m
kube-system      ebs-csi-controller-5c9db44f4f-6twcq        5/5     Running   0          15m
kube-system      ebs-csi-node-ctcfp                         3/3     Running   0          15m
kube-system      ebs-csi-node-mh8w2                         3/3     Running   0          13m
kube-system      kube-proxy-gsw28                           1/1     Running   0          15m
kube-system      kube-proxy-wkz7d                           1/1     Running   0          13m
kube-system      metrics-server-7cc78958fc-n6jrp            1/1     Running   0          16m
projectsveltos   sveltos-agent-manager-67d6ffbd86-5vx9z     1/1     Running   0          15m

Re-enabling both services again

MANAGEMENT CLUSTER TARGET CLUSTER

We see that the ClusterSummary object again shows the list of helmCharts:

~ kubectl -n hmc-system get clustersummary hmc-system-wali-aws-dev-capi-wali-aws-dev -o yaml
apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterSummary
metadata:
  generation: 4
  . . .
  resourceVersion: "12277"
  uid: 500b3a9b-8bf7-479a-a1e9-7edaf314e9dc
spec:
  clusterName: wali-aws-dev
  clusterNamespace: hmc-system
  clusterProfileSpec:
    clusterSelector:
      matchLabels:
        helm.toolkit.fluxcd.io/name: wali-aws-dev
        helm.toolkit.fluxcd.io/namespace: hmc-system
    continueOnConflict: false
    helmCharts:
    - chartName: kyverno
      chartVersion: 3.2.6
      helmChartAction: Install
      . . .
      registryCredentialsConfig:
        plainHTTP: true
      releaseName: kyverno
      releaseNamespace: kyverno
      repositoryName: kyverno
      repositoryURL: oci://hmc-local-registry:5000/charts
    - chartName: ingress-nginx
      chartVersion: 4.11.0
      helmChartAction: Install
      . . .
      registryCredentialsConfig:
        plainHTTP: true
      releaseName: ingress-nginx
      releaseNamespace: ingress-nginx
      repositoryName: ingress-nginx
      repositoryURL: oci://hmc-local-registry:5000/charts
      values: |
        fullnameOverride: ingress-nginx
    reloader: false
    stopMatchingBehavior: WithdrawPolicies
    syncMode: Continuous
    tier: 100
  clusterType: Capi
status:
  dependencies: no dependencies
  featureSummaries:
  - featureID: Helm
    hash: 8ZDFC0FQZ2j1VHZPAeecZpAtyOUBotyUGBJGosO4tYA=
    lastAppliedTime: "2024-09-20T19:08:52Z"
    status: Provisioned
  helmReleaseSummaries:
  - releaseName: kyverno
    releaseNamespace: kyverno
    status: Managing
    valuesHash: Eq4yyx7ALQHto1gbEnwf7jsNxTVy7WuvI5choD2C4SY=
  - releaseName: ingress-nginx
    releaseNamespace: ingress-nginx
    status: Managing
    valuesHash: qYgUi/xTJIMlaXCLxb/XjCBv5xso8nVHHQ0copZdxl4=

Both ingress-nginx and kyverno have again been installed on the target cluster:

~ kubectl get pod -A
NAMESPACE        NAME                                             READY   STATUS    RESTARTS   AGE
ingress-nginx    ingress-nginx-controller-5bfc858768-dmt84        1/1     Running   0          55s
kube-system      aws-cloud-controller-manager-fjfg2               1/1     Running   0          18m
kube-system      calico-kube-controllers-695f6448bd-fckbc         1/1     Running   0          19m
kube-system      calico-node-7tv5t                                1/1     Running   0          18m
kube-system      calico-node-wkxvg                                1/1     Running   0          17m
kube-system      coredns-6997b8f8bd-f966x                         1/1     Running   0          16m
kube-system      coredns-6997b8f8bd-ht4qs                         1/1     Running   0          16m
kube-system      ebs-csi-controller-5c9db44f4f-5cs6w              5/5     Running   0          19m
kube-system      ebs-csi-controller-5c9db44f4f-6twcq              5/5     Running   0          19m
kube-system      ebs-csi-node-ctcfp                               3/3     Running   0          18m
kube-system      ebs-csi-node-mh8w2                               3/3     Running   0          17m
kube-system      kube-proxy-gsw28                                 1/1     Running   0          18m
kube-system      kube-proxy-wkz7d                                 1/1     Running   0          17m
kube-system      metrics-server-7cc78958fc-n6jrp                  1/1     Running   0          19m
kyverno          kyverno-admission-controller-776987899-qw8g6     1/1     Running   0          67s
kyverno          kyverno-background-controller-86b9f95c96-8nmt5   1/1     Running   0          67s
kyverno          kyverno-cleanup-controller-7bbfc97569-zg86g      1/1     Running   0          67s
kyverno          kyverno-reports-controller-665ccb5b65-jg4xb      1/1     Running   0          67s
projectsveltos   sveltos-agent-manager-67d6ffbd86-5vx9z           1/1     Running   0          19m

Finally deleting the ManagedCluster object

~ khmc delete managedclusters.hmc.mirantis.com wali-aws-dev
managedcluster.hmc.mirantis.com "wali-aws-dev" deleted

Wait for a while for the delete to finish . . .

~ kubectl -n hmc-system get managedclusters.hmc.mirantis.com 
No resources found in hmc-system namespace.

We can see that that the associated ClusterProfile and ClusterSummary objects have also been deleted:

~ kubectl get clusterprofiles.config.projectsveltos.io 
No resources found
➜  ~ kubectl -n hmc-system get clustersummaries.config.projectsveltos.io 
No resources found in hmc-system namespace.

@wahabmk wahabmk force-pushed the reconcile-servicetemplates branch 13 times, most recently from 01a018f to 30c73ee Compare September 23, 2024 16:15
@wahabmk wahabmk force-pushed the reconcile-servicetemplates branch 3 times, most recently from ae51ffc to 2bd0085 Compare September 25, 2024 11:16
@wahabmk wahabmk force-pushed the reconcile-servicetemplates branch from 2bd0085 to 3ffa70f Compare September 26, 2024 14:27
dependabot bot and others added 4 commits September 26, 2024 17:57
Bumps [github.com/fluxcd/source-controller/api](https://github.com/fluxcd/source-controller) from 1.3.0 to 1.4.1.
- [Release notes](https://github.com/fluxcd/source-controller/releases)
- [Changelog](https://github.com/fluxcd/source-controller/blob/main/CHANGELOG.md)
- [Commits](fluxcd/source-controller@v1.3.0...v1.4.1)

---
updated-dependencies:
- dependency-name: github.com/fluxcd/source-controller/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
…ithub.com/fluxcd/source-controller/api-1.4.1

Bump github.com/fluxcd/source-controller/api from 1.3.0 to 1.4.1
Kshatrix and others added 10 commits September 27, 2024 18:58
Bumps [github.com/fluxcd/helm-controller/api](https://github.com/fluxcd/helm-controller) from 1.0.1 to 1.1.0.
- [Release notes](https://github.com/fluxcd/helm-controller/releases)
- [Changelog](https://github.com/fluxcd/helm-controller/blob/main/CHANGELOG.md)
- [Commits](fluxcd/helm-controller@v1.0.1...v1.1.0)

---
updated-dependencies:
- dependency-name: github.com/fluxcd/helm-controller/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
* removed/unexported a couple of
  variables being not used as exported
* decreased cognitive burden removing
  redundant fields/smell patterns
* a couple of comments to reflect on
  the public API
…ithub.com/fluxcd/helm-controller/api-1.1.0

Bump github.com/fluxcd/helm-controller/api from 1.0.1 to 1.1.0
Fix CP providers label in charts meta
@wahabmk wahabmk force-pushed the reconcile-servicetemplates branch 2 times, most recently from b00ae13 to 4e0f8c0 Compare September 30, 2024 15:24
@wahabmk wahabmk force-pushed the reconcile-servicetemplates branch from 4e0f8c0 to dadff0a Compare September 30, 2024 15:31
@wahabmk wahabmk merged commit 0e7867b into main Oct 2, 2024
3 checks passed
@wahabmk wahabmk deleted the reconcile-servicetemplates branch October 4, 2024 15:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants